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Peer Review: OMB’s Proposed and Revised Bulletins 



Summary 

In September 2003, the Office of Management and Budget (0MB) published 
a proposed bulletin on “Peer Review and Information Quality” in the Federal 
Register that sought to establish a process by which all “significant regulatory 
information” would be peer reviewed. The scope of the proposed bulletin was very 
broad, covering virtually all agencies and defining regulatory information as “any 
scientific or technical study that ... might be used by local, state, regional, federal 
and/or international regulatory bodies.” Such information would be subject to peer 
review if the agency could determine that it could have a “clear and substantial 
impact on important public policies or important private sector decisions” when 
disseminated. The proposed bulletin placed additional peer review requirements on 
“especially significant regulatory information,” and said agencies were required to 
notify 0MB in advance of any studies that might require peer review and how any 
such reviews would be conducted. 

The proposed bulletin aroused controversy, with some observers expressing 
concern that it could create a centralized peer review system within 0MB that would 
be vulnerable to political manipulation or control by regulated entities. 0MB 
received nearly 200 comments on the proposal, and published a “substantially 
revised” peer review bulletin in April 2004 that was broader in scope than the 
proposed bulletin in that it applied to “influential scientific information” (which 
includes, but is not limited to, regulatory information) and “highly influential 
scientific assessments.” However, agencies were given substantial discretion to 
decide whether information is “influential” and therefore requires a peer review. The 
revised bulletin also allowed agencies to use the National Academy of Sciences for 
peer reviews or to use other procedures that had been approved by 0MB. It also 
provided exemptions for certain classes of information, such as information related 
to national security, products by government-funded scientists that are not 
represented as views of a federal agency, and routine statistical information. 
However, 0MB retained significant authority to decide when information is “highly 
influential” (and, therefore, requires more specific peer review procedures) and to 
approve alternative peer review procedures. 

0MB received more than 50 comments on the revised peer review bulletin, 
many of which were supportive of the changes made to the proposal. However, some 
commenters believed the changes did not go far enough, while others believed that 
0MB had significantly weakened the bulletin. In January 2005, 0MB published a 
final version of the bulletin with what it described as “minor revisions” to the version 
published in April 2004 (e.g., requiring agencies to disclose the identities of peer 
reviewers and to prepare an annual report on their peer review activities). A number 
of issues regarding the implementation of the bulletin remain unclear (e.g. , how much 
discretion agencies will be given to decide when and what kind of peer review is 
required). This report will be updated when any further revisions to the bulletin are 
published or other significant events occur. 
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Peer Review: OMB’s Proposed and 
Revised Bulletins 



On September 15, 2003, the Office of Management and Budget (0MB) in 
coordination with the Office of Science and Technology Policy (OSTP), published 
a proposed bulletin in the Federal Register on “Peer Review and Information 
Quality.” The bulletin, if made final, would have established a process by which all 
“significant regulatory information” and “especially significant regulatory 
information” would be peer reviewed.^ 0MB described the term “peer review” in 
this context as “a scientifically rigorous review and critique of a study’s methods, 
results, and findings by others in the field with requisite training and expertise.” The 
proposed bulletin placed additional peer review requirements on “especially 
significant regulatory information,” and said agencies were required to notify 0MB 
in advance of any studies that might require peer review. The scope of the proposed 
bulletin was very broad, covering virtually all agencies and defining “regulatory 
information” as “any scientific or technical study that ... might be used by local, 
state, regional, federal and/or international regulatory bodies.” 0MB indicated it was 
issuing the bulletin because agencies’ peer review practices were inconsistent, and 
government- wide standards for peer review would make regulatory science more 
competent and credible. 

The proposed bulletin aroused substantial controversy, with some observers 
expressing concern that it could create a centralized peer review system within 0MB 
that would be vulnerable to political manipulation or control by regulated entities. 
0MB received nearly 200 comments on the proposal, including comments from 
Members of Congress, trade associations, public interest groups, and recognized 
experts in the field of peer review and scientific research. As a result of those 
comments, 0MB (again in consultation with OSTP) published what it described as 
a “substantially revised” peer review bulletin in the Federal Register on April 28, 
2004.^ In some ways, the revised bulletin was broader than its predecessor. For 
example, instead of focusing on “significant” and “especially significant regulatory 



* Office of Management and Budget, Executive Office of the President, “Proposed Bulletin 
on Peer Review and Information Quality,” 68 Federal Register (Sept. 15, 2003). This 

proposed bulletin had been released to the public via 0MB ’ s website on Aug. 29 , 2003 . T o 
view a copy, see [http://www.whitehouse.gov/omb/inforeg/peer_review_and_info_quality. 
pdf). 

^ Office of Management and Budget, Executive Office of the President, “Revised 
Information Quality Bulletin on Peer Review,” 69 Federal Register 23230 (Apr. 28, 2004). 
This revised bulletin had been released to the public via OMB’s website on April 15, 2004; 
see [http://www.whitehouse.gOv/omb/inforeg/peer_review041404.pdf]. In this report, the 
first draft of the bulletin is referred to as the “proposed bulletin” and the second draft as the 
“revised bulletin.” Unless otherwise specified, each respective reference includes the 
introductory supplemental information as well as the body of the bulletin per se. 
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information,” the revised bulletin centered on “influential scientific information” 
(which includes, but is not limited to, regulatory information) and “highly influential 
scientific assessments.” In other ways, though, the revised bulletin was less inclusive 
and directive. For example, it gave agencies more discretion to determine when 
information required a peer review, and when the more detailed review requirements 
for “highly influential” information were applicable. Also, unlike the proposed 
bulletin, the revised bulletin did not exclude individuals from being peer reviewers 
if they had received research grants from the agency disseminating the information 
being peer reviewed. 0MB again requested comments from the public on the 
bulletin. 

On December 15, 2004, 0MB published a final version of the peer review 
bulletin on its website.^ The final bulletin was published in the Federal Register on 
January 14, 2005."^ 0MB said this version reflects “minor revisions” made in 
response to more than 50 comments from the public on the revised bulletin. For 
example, the final bulletin requires agencies to disclose the names of peer reviewers 
to the public and adds an annual reporting requirement to allow 0MB to track how 
agencies are using the bulletin. However, agencies are still afforded substantial 
discretion to determine when and what type of peer review is required. 0MB also 
retains substantial discretion in certain areas. 

This report briefly describes 0MB ’s September 2003 peer review proposal, the 
concerns that were raised about that proposal in the public comments and elsewhere, 
and 0MB ’s April 2004 revised bulletin. It then describes comments on the revision 
and 0MB ’s final bulletin. First, however, the report provides some background on 
peer review, particularly in the context of regulatory policy and recent efforts to 
improve the quality of regulatory information. 



Peer Review and Regulatory Policy 

There is no official, government- wide definition of “peer review,” even in the 
0MB bulletins.^ Agency-specific definitions and descriptions of the term differ, but 
all of them contain the basic concept of a review of scientific or technical merit by 
individuals with both sufficient technical competence and no unresolved conflicts of 



^ Office of Management and Budget, Final Information Quality Bulletin for Peer Review, 
Dec. 15, 2004, at [http://www.whitehouse.gov/omb/inforeg/peer2004/peer_bulletin.pdf]. 

Office of Management and Budget, Final Information Quality Bulletin for Peer Review, 
70 Federal Register 2664 (Jan. 14, 2005). 

^ While the term “peer review” is not explicitly defined in 0MB ’s bulletins, the 
supplementary information in the final bulletin describes peer review generally as “one of 
the important procedures used in science to ensure that the quality of published information 
meets the standards of the scientific community. It is a form of deliberation involving an 
exchange of judgments about the appropriateness of methods and the strength of the author’ s 
inferences. Peer review occurs when a draft product is reviewed for quality by specialists 
who were not involved in producing the draft.” Arguably, the methods laid out in the 
bulletin for conducting peer review could also be considered an implicit definition of the 
concept. 
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interest. In its final bulletin, 0MB states that peer review “typically evaluates the 
clarity of hypotheses, the validity of research design, the quality of the data collection 
procedures, the robustness of the methods employed, the appropriateness of the 
methods for the hypotheses being tested, the extent to which the conclusions follow 
from the analysis, and the strengths and limitations of the overall product.”^ 

Support for Peer Review 

Some type of peer review has been used for centuries within the scientific 
community to judge the quality of science. Peer review can take many different 
forms, and is used for a variety of purposes. For example, it is used commonly by 
federal agencies to evaluate research proposals, and plays a major role in funding 
decisions. In these cases it is often part of a broader category of evaluation known 
as merit review. Peer review is also the usual method by which the editors of 
scientific publications evaluate proposed research reports. It even plays an important 
role in many research institutions, including many government agencies, in decisions 
about retention and promotion for individual scientists and in reviewing research 
programs . A February 1 999 report from the Committee on Science, Engineering, and 
Public Policy concluded that expert review (the most common form of which is peer 
review) is the most effective means of evaluating federally funded research 
programs.^ 

Peer review is also used for scientific and technical products relating to policies, 
including regulations, to determine whether the underlying scientific findings are 
well supported. For example, such peer review was established as Environmental 
Protection Agency (EPA) policy in 1993.* In 1996, a panel of leading economists 
concluded that peer review should be used for economic analyses supporting 
regulations with a potentially large impact on the economy (e.g., those whose annual 
economic costs exceed $1 billion).^ The panel also indicated that reviewers should 
be selected based on their demonstrated expertise and reputations. 

In its peer review bulletins, 0MB recognized the variety of ways that peer 
review is used. 0MB said that independent peer review is especially important in the 
regulatory arena because federal agencies often develop or fund the science that 
underlies their regulations, and then oversee the peer review of those scientific 
studies — thereby creating the appearance of a conflict of interest. 



® Final Information Quality Bulletin for Peer Review, p. 2. 

^ The Committee on Science, Engineering, and Public Policy, Evaluating Federal Research 
Programs: Research and the Government Performance and Results Act (Washington: 
National Academies Press, Feb. 1999). The Committee is a joint committee of the National 
Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. 

* U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal 
Science Agencies Vary, GAO/RCED-99-99, Mar. 1999. 

^ Kenneth J. Arrow, et al, Benefit-Cost Analysis in Environmental, Health, and Safety 
Regulation: A Statement of Principles (Annapolis, MD: AEI Press, 1996). 
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Legislative Proposals. A number of statutes and legislative proposals have 
advocated the use of peer review in a regulatory context for particular issues. For 
example, the Safe Drinking Water Act (42 U.S.C. 300g- 1(b)(3)(A)) requires EPA, 
when taking action under the act based on science, to use the “best available, peer 
reviewed science and supporting studies.” In the 108* Congress, the Water 
Resources Development Act of 2003 (H.R. 2557) would have generally required 
project studies to be subject to peer review by an independent panel of experts if the 
project has an estimated total cost of more than $50 million. Similarly, the Sound 
Science for Endangered Species Act Planning Act of 2003 (H.R. 1662, 108* 
Congress) would have directed the Secretary of the Interior, in making decisions 
about species protection, to give greater weight to certain kinds of data that had been 
peer reviewed by qualified individuals as defined in the bill. 

There have also been legislative efforts to require peer review more broadly. 
Eor example, in the 106* Congress, the Senate considered but did not enact bipartisan 
legislation (S. 746) that would have required virtually all agencies to provide for an 
independent peer review of any required risk assessments and cost-benefit analyses 
of major rules that the agencies or the 0MB Director reasonably anticipated were 
likely to have a $500 million effect on the economy. The bill would have required 
that peer reviews be conducted through panels that were “broadly representative” and 
involved participants with relevant expertise who were “independent of the agency.” 
However, if an agency certified that adequate peer review had already been 
conducted, and the 0MB Director agreed, no further peer review would have been 
required. In its comments on this legislation, the General Accounting Office (GAO) 
(now the Government Accountability Office) generally supported the use of peer 
review in this context, noting that “the rigorous, independent review of economic 
analyses should help enhance the quality, credibility, and acceptability of agencies’ 
decisionmaking.”^® However, GAO cautioned that (given the number of reviews 
contemplated) agencies would need to plan carefully for the reviews, and that the 
panels would need to reflect all points of view. 

Presidential Support. Recent presidential administrations have also 
supported peer review as a preferred means of assessing scientific research, both 
prospectively and retrospectively. Eor example, beginning with the EY 1996 budget 
cycle, 0MB and OSTP have jointly provided annual direction to agencies, 
encouraging them to emphasize the funding of peer-reviewed research over nonpeer- 
reviewed research for most scientific activities. In 1997, the then- Administrator of 
0MB ’s Office of Information and Regulatory Affairs (OIRA) testified that the 
Clinton Administration supported peer review, but also said the administration 
recognized that it is not cost-free in terms of agencies’ resources or time." 

On September 20, 2001, the new OIRA Administrator for the George W. Bush 
Administration issued a memorandum for the President’s Management Council 
recommending (among other things) that agencies subject regulatory impact analyses 



U.S. General Accounting Office, Regulatory Reform: Comments on S. 746 — The 
Regulatory Improvement Act of 1999, GAO/T-GGD/RCED-99-163, Apr. 1999, p. 6. 

" Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on 
Governmental Affairs, Sept. 12, 1997. 
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and supporting technical documents for “economically significant” and “major” rules 
(e.g., those with a $100 million annual effect on the economy) to independent, 
external peer review. The OIRA Administrator also recommended certain criteria 
for peer review (e.g., disclosure by peer reviewers of prior technical or policy 
positions on the issues at hand and their sources of personal and institutional 
funding), and said OIRA would give agency analyses that had undergone such a 
review “a measure of deference” during its reviews of their regulatory proposals 
under Executive Order 12866.^^ 

Cautionary Notes About Peer Review 

Although the concept of peer review has generally received wide support, some 
observers have also raised cautionary notes or mentioned certain limitations to the 
approach in particular situations. For example, as 0MB indicated in its 1997 
testimony, peer review costs can be significant for agencies in terms of both time and 
agency resources. To pay for peer review procedures, agencies may have to divert 
resources from other areas (e.g., regulatory enforcement or standards development). 
Scientists from academic institutions who perform peer reviews on a voluntary basis 
may also incur opportunity costs with respect to other activities such as teaching and 
research. Other concerns have focused on how peer reviews have been implemented. 
For example, GAO noted that peer review methods varied within and among federal 
agencies, and that agencies’ economic analyses of major rules were often not peer 
reviewed. A number of observers have expressed concerns about the impact of peer 
review requirements on the pace of regulatory activity, with additional requirements 
exacerbating what is already often regarded as an “ossified” rulemaking process.'^ 



For a copy of this memorandum, see 
[http://www.whitehouse.gov/omb/inforeg/oira_review-process.html]. 

Executive Order 12866, “Regulatory Planning and Review,” 58 Federal Register 51135, 
Oct. 4, 1993. For a description of 0MB ’s reviews under this executive order, see CRS 
Report RL32397, Federal Rulemaking: The Role of the Office of Information and 
Regulatory Affairs, by Curtis W. Copeland. 

Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on 
Governmental Affairs, Sept. 12, 1997. 

GAO/RCED-99-99. For example, GAO reported that some agencies conducted peer 
reviews of research proposals primarily by mail, while others generally relied on panels or 
committees. All agencies used a combination of external and internal reviewers for these 
reviews, but one (Federal Aviation Administration) relied primarily on agency employees 
who were not employed in the project but had the required expertise. 

U.S. General Accounting Office, Regulatory Reform: Agencies Could Improve 
Development, Documentation, and Clarity of Regulatory Economic Analyses, GAO/RCED- 
98-142, May 1998. Only 1 of the 20 economic analyses that GAO reviewed had been 
independently peer reviewed. 

See, for example, Thomas O. McGarity, “Some Thoughts on ‘Deossifying’ the 
Rulemaking Process, Duke Law Journal, vol. 41 (June 1992), pp. 1385-1462; Richard J. 
Pierce, Jr., “Seven Ways to Deossify Agency Rulemaking,” Admmwtradve Law Review, vol. 
47 (winter 1995), pp. 59-93; Paul R. Verkuil, “Rulemaking Ossification — A Modest 

(continued...) 
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In fact, some critics have suggested that regulatory relief and delay is the primary 
purpose of peer review proposals in this context. 

Still other concerns about peer review have centered on issues of bias and 
balance. Experts agree that effective peer review panels must be (and must be 
perceived to be) free from any significant conflict of interest and properly balanced, 
allowing for a wide range of views and appropriate expertise. However, in June 
2001, GAO reported that the policies and procedures developed by EPA’s Science 
Advisory Board to ensure balance and independence of the Board’s peer reviewers 
had limitations that reduced their effectiveness.^^ Eor example, GAO said that the 
staff office did not routinely ensure that panel members’ financial disclosures were 
complete or that they contained enough information to determine whether a conflict 
of interest existed. Also, panel members reportedly did not have to disclose 
information regarding previous positions on the matter being reviewed until the 
panel’s first meeting, thereby making it difficult to determine the independence and 
balance of the panel members until after they had been selected. GAO also reported 
that the panel’s policies and procedures did not adequately inform the public about 
the points of view represented on the panels. GAO made several recommendations 
in this report that were designed to better ensure that the Science Advisory Board’s 
peer review panels are independent and balanced. 

Procedural Flexibility or Uniformity. As noted previously, GAO reported 
that federal agencies’ peer review practices are not consistent. However, several 
studies of peer review in the federal government have suggested that rigidly uniform 
peer review procedures may not be desirable. Eor example, a 1996 National Science 
and Technology Council report indicated that peer review implementation should be 
flexible and “appropriate to the nature of scientific processes.”^® Similarly, in its 
1 999 report on peer review, GAO reported that OSTP believed agencies ’ peer review 
practices should be “flexible and tailored to agency missions and type of research, 
and that specific uniform practices should not be dictated for every agency or all 
federally funded research.”^^ GAO also said that agencies viewed a variety of peer 
review methods as “both appropriate and essential, reflecting the varying nature of 
the research and its purposes, the differences in research timelines, the broad 
spectrum of [research and development] performers, and the varying funding 



(...continued) 

Proposal,” Admmwtradvc Law Review, vol. 47 ( summer 1995), pp. 453-459. 

U.S. General Accounting Office, EPA’s Science Advisory Board Panels: Improved 
Policies and Procedures Needed to Ensure Independence and Balance, GAO-01-536, June 
2001 . 

GAO/RCED-99-99. 

Subcommittee on Research, Committee on Fundamental Science, National Science and 
Technology Council, Assessing Eundamental Science (Washington: National Science 
Foundation, July 1996). The National Science and Technology Council was established by 
executive order in November 1993 to coordinate federal research and development, and to 
establish clear national goals for science and technology investments. To view a copy of this 
report, see [http://www.nsf.gov/sbe/srs/ostp/assess/start.htm]. 

GAO/RCFD-99-99, p. 5. 
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mechanisms, such as grants, contracts, and cooperative agreements.” Notably, 
although GAO pointed out that peer review practices varied across and within 
agencies, it did not recommend greater uniformity in those practices. 

Peer Review and the Information Quality Act 

OMB said that the final peer review bulletin was being issued “under the 
Information Quality Act [IQA] and OMB’s general authorities to oversee the quality 
of agency information, analyses, and regulatory actions. The IQA (also known as 
the “Data Quality Act”) was enacted in December 2000 as Section 515 of the 
Treasury and General Government Appropriations Act for Fiscal Year 2001 (P.L. 
106-554). The act supplemented the Paperwork Reduction Act (PRA), which already 
required OMB to “develop and oversee the implementation of policies, principles, 
standards, and guidelines to apply to federal agency dissemination of public 
information.”^^ The PRA also required agencies to manage their information 
resources to “improve the integrity, quality, and utility of information to all users 
within and outside the agency.”^"^ 

Although little noticed at the time it was enacted, the IQA has subsequently 
been the subject of significant debate and controversy.^^ The act required OMB to 
issue guidance to federal agencies designed to ensure the “quality, objectivity, utility, 
and integrity” of information disseminated to the public. It also required agencies to 
issue their own information quality guidelines, and to establish administrative 
mechanisms that allow affected persons to seek correction of information maintained 
and disseminated by the agencies that does not comply with the OMB guidance. 
However, it did not require either OMB or the agencies to issue peer review bulletins. 

OMB published its final government-wide IQA guidelines in the Federal 
Register in February 2002.^^ In them, OMB encouraged (but did not require) the use 
of peer reviews in the development of agency-disseminated information. 
Specifically, OMB indicated that agencies can presume that data are sufficiently 
“objective” if they have been subject to an independent peer review process (e.g., as 
used by scientific journals), but said a member of the public could rebut this 
presumption “based on a persuasive showing by the petitioner in a particular 
instance.” OMB also indicated that journal peer review may not be sufficient for 



Office of Management and Budget, Final Information Quality Bulletin for Peer Review , 
op. cit., p. 2666. 

44 U.S.C. 3504 (d)(1). 

44 U.S.C. 3506(b)(1)(C). 

For a more complete discussion of the IQA and OMB’s information quality guidelines, 
see CRS Report RL32532, The Information Quality Act: OMB’s Guidance and Initial 
Implementation, by Curtis W. Copeland and Michael Simpson. 

Office of Management and Budget, “Guidelines for Ensuring and Maximizing the Quality, 
Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies,” 67 
Federal Register 8452, Feb. 22, 2002. OMB published proposed guidelines in June 2001 
(66 Federal Register 34489), and published an earlier version of the final guidelines in 
September 2001 (66 Federal Register 4911%). 
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information likely to have an important public policy or private sector impact. 
Finally, the IQA guidelines set minimum standards for the transparency of agency- 
sponsored peer review (referencing the standards in the above-mentioned September 
20, 2001, memorandum issued by the OIRA Administrator — e.g., disclosure of 
reviewers’ prior technical or policy positions and sources of funding). 

The final peer review bulletin also cited Executive Order 12866 and 0MB ’s 
authority to “manage the agencies under the purview of the President’ s Constitutional 
authority to supervise the unitary Executive Branch” as other authorities supporting 
the bulletin’s issuance.^^ E.O. 12866 says that OIRA within 0MB is the “repository 
of expertise concerning regulatory issues,” and says that agencies should base their 
decisions on “the best reasonably obtainable scientific, technical, economic, or other 
information concerning the need for, and consequences of the intended regulation.”^* 
However, the executive order does not specifically mention peer review. 



OMB’s Proposed Peer Review Bulletin 

When it published its proposed peer review bulletin in the Federal Register in 
September 2003, 0MB noted in the preamble to the bulletin the importance of peer 
review and the sometimes inconsistent nature of its application. Eor these reasons, 
the office proposed requiring peer review for “the most important scientific and 
technical information relevant to regulatory policies that [agencies] disseminate to 
the public. 0MB said the new requirements would supplement, but not supplant, 
both its own information quality guidelines and any agency-specific peer review 
requirements for regulatory information. Although 0MB noted that its bulletins “are 
not laws or binding regulations,” as a practical matter, 0MB ’s instructions to the 
agencies carry substantial weight because of its authority to review agencies’ budget 
proposals and most significant regulations before they are published. 

In essence, 0MB ’ s bulletin proposed that each covered federal agency take three 
actions: (1) have all “significant regulatory information” that the agencies intend to 
disseminate peer reviewed; (2) have “especially significant regulatory information” 
peer reviewed according to even higher standards; and (3) provide 0MB with 
information at least once each year about upcoming scientific studies that might 
support significant regulatory disseminations, and the agencies’ plans for conducting 
peer reviews of those studies. The proposed bulletin also contained several other, 
related requirements. The bulletin defined “agency” broadly, including cabinet 
departments and agencies, independent agencies (e.g., EPA), and independent 
regulatory agencies (e.g., the Eederal Communications Commission and the Nuclear 
Regulatory Commission). 



Office of Management and Budget, Final Information Quality Bulletin for Peer Review, 
op. cit., p. 7. 

Executive Order 12866, section 1(b)(7). 

68 Federal Register 54026, Sept. 15, 2003. 
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Peer Review of Significant Reguiatory Information 

The proposed bulletin defined “significant regulatory information” as scientific 
or technical information that qualifies as “influential” under 0MB ’s information 
quality guidelines and is “relevant to regulatory policies.” OMB’s information 
quality guidelines define “influential information” as information that the agency can 
“reasonably determine” will have a “clear and substantial impact on important public 
policies or important private sector decisions” when it is disseminated to the public. 
The proposed peer review bulletin said information is “relevant to regulatory 
policies” if it “might be used by local, state, regional, federal and/or international 
regulatory bodies.” The bulletin indicated that the peer review for this type of 
information should be “appropriate” and “scientifically rigorous,” but also said the 
appropriate peer review mechanism would depend on “the novelty and complexity 
of the science to be reviewed, the benefit and cost implications, and any controversy 
regarding the science.” 

For information at the lower end of those dimensions, the proposed bulletin 
indicated that a review by qualified specialists within a separate part of the agency 
disseminating the information could satisfy the peer review requirement. Also (as 
in the information quality guidelines), agencies were allowed to presume that any 
studies that had already been independently peer reviewed did not have to be 
reexamined, but the bulletin said this presumption “is rebuttable based on a 
persuasive showing in a particular instance.” Finally, the bulletin said peer review 
was not needed for significant regulatory information related to national defense or 
foreign affairs, or information that is disseminated in the course of an individual 
agency adjudication or proceeding on a permit application. 

Review of Especially Significant Regulatory Information 

The proposed bulletin indicated that regulatory information should be 
considered “especially significant” (and therefore subject to more specific peer 
review requirements) if (a) the agency intends to disseminate it in support of an 
economically significant regulatory action (e.g., with a $100 million annual impact 
on the economy),^” (b) its dissemination would otherwise have a clear and substantial 
effect on important public policies or important private sector decisions (with an 
impact of $100 million in any year), or (c) 0MB determines it is of “significant 
interagency interest” or “is relevant to an Administration policy priority.” The 
bulletin laid out a series of proposed requirements for these types of peer reviews. 
For example, it said agencies must 

• select peer reviewers primarily based on their scientific and technical 
expertise and “strive” to appoint reviewers who are independent of 
the agency and do not have any “real or perceived conflicts of 
interest” (e.g., have no financial interest in the issue, do not seek or 
receive substantial funding from the agency, and have not conducted 
a peer review for the agency or advocated a position in recent years 



The definition of an “economically significant” regulatory action can be found in section 
3(f)(1) of Executive Order 12866. 
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on the specific topic being examined). If obtaining the requisite 
expertise requires the selection of a panel member with a perceived 
bias, the bulletin said agencies must “ensure that another reviewer 
with a contrary bias is appointed to balance the panel”; 

• provide peer reviewers with an explicit statement describing the 
purpose and scope of the review, and sufficient information to 
understand the data, methods, and conclusions of the material being 
reviewed; 

• permit other interested agencies and persons to submit comments 
and provide those comments to peer reviewers for their 
consideration; and 

• direct peer reviewers to issue a final report describing the nature of 
their review and their findings and conclusions, including any 
dissenting opinions. The report must also describe the names, 
affiliations, and qualifications of all peer reviewers and any prior 
affiliations with the agency. 

Also, agencies were instructed to consult with 0MB and OSTP regarding the 
“sufficiency” of their planned peer review policies and, on request, to discuss with 
0MB the sufficiency of their plans to review specific documents. 

0MB Notification of Upcoming Peer Reviews 

The proposed bulletin indicated that, at least once each year, each agency must 
provide 0MB with a summary description of any “existing, ongoing, or 
contemplated” studies that might support significant regulatory information that the 
agency intends to disseminate within the next year. Each agency was also to provide 
0MB with the agency’s plan for conducting peer reviews of such studies under the 
bulletin’ s requirements, including an agency contact person. 0MB indicated that, to 
reduce paperwork, this information was to be included in other reports being 
submitted to 0MB under Executive Order 12866 or the Information Quality Act. 

Other Provisions 

The proposed bulletin also contained several other peer review requirements or 
provisions, including the following: 

• Agencies that are likely to disseminate “significant” or “especially 
significant” regulatory information were required to supplement or 
amend their information quality guidelines to incorporate the 
requirements of the proposed peer review bulletin for “significant” 
and “especially significant” information. These agencies’ 
supplements or amendments were also to address (a) what factors 
could preclude an individual’s participation as a peer reviewer, (b) 
how confidential business information and other confidential 
information would be protected, and (c) what information about peer 
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reviewers should be disclosed (e.g., sources of funding that could 
suggest a conflict of interest). 

• 0MB could waive the requirements for peer review if an agency 
made “a compelling case” that a waiver was necessary (e.g., an 
imminent health hazard or homeland security threat). 

• Agencies were permitted to retain an outside firm to oversee the peer 
review process to select and supervise the peer review panels, which 
the bulletin indicated would allow the agencies to avoid the 
requirements of the Federal Advisory Committee Act (FACA).^' 

0MB initially said these peer review requirements would apply to information 
disseminated on or after January 1 , 2004. 0MB initially requested public comments 
on the proposed bulletin by October 28, 2003, but later extended the comment period 
to December 15, 2003, and then to January 16, 2004. 



Comments on the Proposed Peer Review Bulletin 
and OMB’s Responses 

OMB said that it received 187 comments on the proposed bulletin on peer 
review, and posted those comments on its website. In addition to this formal 
commenting process, OMB and OSTP sponsored an open workshop on the peer 
review requirements in November 2003 The workshop was hosted by the National 
Academy of Sciences (NAS), which subsequently endorsed the bulletin’s objectives 
but expressed several concerns. OMB said the comments that it received varied 
substantially, with some highly favorable and others suggesting that the bulletin be 
withdrawn and reconsidered. In general, supporters of the proposed bulletin 
reportedly indicated that peer review standards across the government were 
inconsistent, and that more consistent use of peer review could increase the technical 
quality and credibility of regulatory science. They also reportedly asserted that peer 
review could protect science-based regulations from political criticism and litigation. 
For example, the U.S. Chamber of Commerce said that OMB had “provided a 
Bulletin of high quality and soundjudgment,” and strongly supported “OMB’s efforts 
to improve the quality, objectivity, utility, and integrity of information disseminated 



The bulletin cited Byrd v. EPA, 174 F.3d 239 (D.C. Cir. 1999), “holding that peer review 
panels selected and supervised by outside consultants are not governed by [FACA].” For 
more information on federal advisory committees and FACA, see CRS Report RL30260, 
Federal Advisory Committees: A Primer, by Stephanie Smith; or CRS Report RL30795, 
General Management Laws: A Compendium, coordinated by Clinton T. Brass, pp. 35-37. 

To view the comments provided on the proposed peer review bulletin, see [http://www. 
whitehouse.gOv/omb/inforeg/2003iq/iq_list.html]. 

To view a transcript of this conference, see [http://www7.nationalacademies.org/stl/ 
Peer_Review_Transcript.pdf] . 




http://wikileaks.org/wiki/CRS-RL32680 



CRS-12 



by the federal government to the public.”^"^ Similarly, the National Petrochemical & 
Refiners Association strongly supported 0MB ’s proposed bulletin, saying that it 
would “correct the current variability in federal agency implementation of peer 
reviews by providing the needed guidance on process requirements.”^^ 

0MB said that many critics of the proposed bulletin viewed it as an effort to 
inject political considerations into the world of science, and to use the uncertainty 
that inevitably surrounds science as an excuse to delay new rules that could be costly 
to regulated entities. They also reportedly said that the bulletin appeared more 
concerned about peer reviewers’ possible conflicts of interest with agencies than their 
potential conflicts with regulated industries. 

In its summary of the public comments, 0MB identified a number of broad 
themes and noted how it responded to those comments. Specifically, 0MB said the 
comments centered on such issues as the need for the bulletin, its scope and 
applicability, the prescriptive nature of the review process for “especially significant” 
information, and the costs of implementing the bulletin. 

Need for the Bulletin 

OMB said many commenters indicated that it was unclear why the bulletin was 
needed. For example, the American Public Health Association said it was “unaware 
of any evidence that the current system is not working or any examples of 
inappropriate or flawed federal regulations being promulgated as a result of failure 
to peer review.”^^ In response, OMB said it would provide additional information in 
the revised bulletin on the importance of peer review and the variability of agencies’ 
peer review policies. 

Clarifications of Scope and Applicability 

A number of commenters reportedly said the scope and applicability of the 
bulletin was unclear. 

• Some commenters questioned the focus on “regulatory” information, 
and said it would be difficult to know in advance whether a 
particular study might be used in support of future regulatory action. 

In its response to the comments, OMB said it would revise the 



For a copy of the Chamber’s comments on the proposed bulletin, see [http://www. 
whitehouse.gov/omb/inforeg/2003iq/170.pdf|. 

For a copy of the association’s comments on the proposed bulletin, see [http://www. 
whitehouse.gOv/omb/inforeg/2003iq/l 1 1 .pdf] . 

Office of Management and Budget, “Summary of Public and Agency Comments on 
Proposed Bulletin on Information Quality and Peer Review, Including Responses by OMB,” 
Apr. 15, 2004. A copy of this summary is available at [http://www.whitehouse.gov/omb/ 
inforeg/peer_re view_comment.pdf]. 

Letter to Dr. Margo Schwab, Dec. 11, 2003. To view a copy of this letter, see [http:// 
www.whitehouse.gOv/omb/inforeg/2003iq/86.pdf]. 
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bulletin to focus on “influential” scientific information instead of the 
potential regulatory impact of the information disseminated. As 
noted previously, the IQA guidelines state that information should 
be considered “influential” if “the agency can reasonably determine 
that dissemination of the information will have or does have a clear 
and substantial impact on important public policies or important 
private decisions. 

• Some commenters expressed concern that the criteria for requiring 
the more stringent peer review requirements for highly significant 
information were too broad. In response, 0MB said it would narrow 
the applicability of those requirements to cover only “scientific 
assessments” that either have a $500 million annual effect on 
important public policies or private sector decisions or involve 
novel, complex, or precedent- setting approaches or significant 
interagency interest. 

• 0MB said it agreed with comments questioning the applicability of 
the bulletin’s requirements to agencies’ regulatory impact analyses, 
and said it would more clearly state that those analyses would not be 
covered by the revised bulletin. However, 0MB went on to say 
that the bulletin’s requirements would cover the models and data 
underlying those analyses, and encouraged agencies to have their 
regulatory impact analyses peer reviewed by “government analysts.” 

• 0MB also said it would clarify that the bulletin did not apply to 
products released by government-funded scientists that were not 
represented as the views of the agency supporting the research. 

• Finally, in response to several concerns, 0MB said it would exempt 
from the bulletin’s coverage any time-sensitive medical, public 
health, and safety disseminations, and allow agency heads to waive 
or defer the bulletin’s requirements “where warranted by a 
compelling rationale.” 

Agency Discretion and Peer Review Procedures 

Commenters also said that the proposed bulletin was too prescriptive in the 
procedures applicable to “especially significant” regulatory information, and said 
extensive peer reviews would be costly and could lead to delays in regulatory 
decisions. In response, 0MB said that it would give agencies additional discretion 
in the revised bulletin to determine the level of peer review required for any 
particular document, and to use alternative procedures if they could demonstrate that 
those procedures would meet the requirements in the government- wide and agency- 



Office of Management and Budget, “Summary of Comments.” 

0MB said that guidance for what constitutes a good regulatory impact analysis is in 0MB 
Circular A-4. To view a copy of this circular, see [http://www.whitehouse.gov/omb/ 
circulars/a004/a-4.pdf| . 
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specific IQA guidelines. The response included significant discussion of the current 
and potential role of NAS in conducting peer reviews. In its comments, NAS had 
pointed to the role its committees have played in performing peer reviews of various 
agency programs, activities, policies, and proposals. In determining the appropriate 
level of review, 0MB said agencies should consider the costs and benefits of peer 
review. 0MB also emphasized that its role is not to veto studies, but rather to ensure 
that they have the appropriate level of review before implementation. 

Selection of Reviewers 

OMB also said it was incorporating suggestions regarding the selection of peer 
reviewers, including that agencies solicit reviewers from the public and from 
scientific and professional societies. One major concern of the commenters was that 
the criteria outlined in the proposed bulletin (e.g., discouraging participation of 
reviewers who were seeking or receiving funding via research grants from an agency) 
could result in the most qualified reviewers being excluded from consideration. 
OMB also said that the revised bulletin would make clear that the receipt of research 
grants through peer-reviewed competitions does not disqualify a potential reviewer. 

Public Participation 

According to OMB, many of the commenters raised concerns about public 
participation in the peer review process, questioning (among other things) whether 
that participation would be useful and raising logistical considerations. As a result, 
OMB said it had decided to allow the agencies to decide whether public participation 
would be allowed and in what forms. However, OMB said it would provide for 
public comment on the agencies’ peer review plans for upcoming documents. 

Cost of Implementing the Bulletin 

Finally, OMB said some of the commenters requested that OMB provide an 
estimate of the costs associated with implementation of the peer review bulletin. For 
example, in a March 2004 letter to OMB, Senators Joseph Lieberman and Richard 
Durbin said it was “curious that OIRA — an office known for insisting on cost- 
benefit analysis — provides no estimate of costs, nor documentation of the benefits 
gained or specific harms that will be reduced by the proposal. We suggest that the 
proposed rules do not pass OMB’s own test of good rulemaking.”'^'’ 

OMB estimated that there would be about 1 25 “influential” scientific documents 
covered each year that are related to rulemaking, and at least 1,250 other documents 
each year that are unrelated to rulemaking. OMB said some of these documents may 
be exempt or otherwise not require peer reviews (e.g., they had already been 
adequately peer reviewed), but did not quantify that assertion. OMB said that a “high- 
quality individual letter review” would cost an agency about $5,000 per document. 
Therefore, assuming that 1,000 of the 1,250 documents required peer review and 



To view a copy of this letter, see [http://govt-aff.senate.gov/index.cfm?FuseAction= 
PressReleases.Detail&PressRelease_id=694&Affiliation=R] . This letter was not part of the 
187 comments that OMB listed on the proposed bulletin. 
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could be reviewed by letter, 0MB said the cumulative cost of these reviews would 
be $5 million. 

0MB said “one or two dozen” of these doeuments could be considered “highly 
influential scientific assessments” requiring more intense peer review. 0MB 
estimated that an average peer review of these documents would cost $50,000. 
Therefore, if 20 of these “highly influential” assessments were peer reviewed, the 
cost would be $1 million. 0MB said these costs are “likely large overestimates of 
the incremental costs of the Bulletin because many agencies already engage in peer 
review practices.” The response also included an analysis that provided a 
methodology for agencies to use in deciding when a more expensive peer-review 
study by NAS would be cost-effective."^^ 

Other Comments 

Several other types of comments were offered on the proposed guidelines that 
were not included in 0MB ’s summary. For example: 

• 0MB W atch questioned 0MB ’ s authority to issue a bulletin on peer 
review, noting that the PR A and the IQ A do not explicitly provide 
the office with that responsibility, and that Congress had previously 
considered and rejected several proposals to mandate peer review. 

0MB Watch said that if 0MB believes government-wide peer 
review requirements are needed, “it should instead submit its ideas 
to Congress for consideration and possible authorization.”"^^ 

• The American Bar Association (ABA), noting that the proposed 
bulletin required peer review whenever the impact of the regulatory 
action would have exceeded $100 million, recommended that 0MB 
limit the bulletin’ s categorical requirement for peer review to “those 
situations in which the scientific information is at least arguably 
complex, novel, or controversial.”"^^ The ABA said peer review “is 
simply not the correct mechanism to address significant use of 
routine, established, or accepted scientific information.” 

• The ABA also said it agreed that one factor in determining whether 
an individual should be a peer reviewer is whether that individual is 
receiving or seeking substantial funding from the agency involved. 
However, the ABA also said that an issue of equal concern is 
whether a potential reviewer is receiving or seeking funding from a 
private company or industry or citizen group with a stake in the 
information involved. 



0MB estimated that such a study might cost $1 million and take two years to complete. 

Letter to Dr. Margo Schwab, Dec. 15, 2003. To view a copy of this letter, see [http:// 
w ww. whitehouse. gov/omb/inforeg/2003iq/ 153 .pdf] . 

Letter to Dr. Margo Schwab, Dec. 23, 2004. To view a copy of this letter, see [http:// 
WWW. whitehouse.gOv/omb/inforeg/2003iq/l 86.pdf] . 
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• The American Association for the Advancement of Science (AAAS) 
said it was concerned that one of the bulletin’s factors in selecting 
reviewers — whether the prospective reviewer had advocated a 
position on the matter at issue — could exclude a qualified scientist 
who had simply made statements based on a “preponderance of peer 
reviewed scientific evidence” (e.g., that particulate matter of a 
certain size had been shown to cause asthma in children). AAAS 
said such statements should not be considered “advocacy.” The 
organization also said that 0MB should make clear that individual 
comments in agencies’ peer review reports would not be associated 
with individual reviewers. In March 2004, the AAAS Council 
adopted a resolution strongly opposing the proposed bulletin and 
requesting that it be withdrawn.'*^ 

• The AEI-Brookings Joint Center for Regulatory Studies 
recommended (among other things) that 0MB or some other entity 
conduct an evaluation of the peer review program after a certain 
amount of time “to determine whether peer review actually led to a 
material improvement in the quality of the regulatory analysis.” 
AEI-Brookings said if 0MB could not develop a reasonable 
approach to evaluation, “it probably should not proceed with this 
effort.” 



OMB’s Revised Peer Review Bulletin 

The comments that 0MB received on the proposed bulletin led the office to 
release what it described as a “substantially revised” bulletin in April 2004. The 
changes were characterized in the media as a “partial retreat” in 0MB ’s initiative."^^ 
0MB said the revised bulletin provided a more extensive discussion of why 
government- wide peer review guidance was needed, gave agencies more discretion 
in determining the appropriate type of peer review for specific products, and more 
clearly indicated that the guidance does not create any new rights for litigation. Also, 
the revised bulletin permitted an agency (under certain circumstances) to use a 
scientist as a peer reviewer who had received research grants from the agency — a 
practice that was discouraged in the proposed bulletin. The table appended to this 



Letter to Dr. Margo Schwab, Dec. 12, 2003. To view a copy of this letter, see [http:// 
WWW. whitehouse.gOv/omb/inforeg/2003iq/8 1 .pdf] . 

American Association for tbe Advancement of Science, Council, “On the 0MB Proposed 
Peer Review Bulletin,” Mar. 9, 2004, at [http://archives.aaas.org/docs/documents.php7doc_ 
id=434]. 

Robert W. Habn and Robert E. Litan, Comment on Peer Review and Information Quality, 
Regulatory Analysis 03-11, Dec. 2003. For a copy of this comment, see [http://www.aei. 
brookings.org/admin/authorpdfs/page.php?id=309]. 

Rick Weiss, “0MB Modifies Peer-Review Proposal: Guidelines Partly Retreat from Strict 
Control of Agencies’ Information Process,” Washington Post, Apr. 16, 2004, p. A-19. 
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report provides a side-by-side comparison of the proposed and revised bulletins. The 
sections below discuss similarities and differences in the documents. 

Changes in Scope and Application 

Among the more notable changes made to the bulletin were revisions to the 
scope and applicability of the peer review requirements. In general, the scope of the 
revised bulletin was arguably much broader than the proposed bulletin. For example, 
whereas the proposed bulletin applied to “significant regulatory information,” the 
revised bulletin applied to “influential scientific information” — a scope that 
included, but was not limited to, regulatory information."^* However, the revised 
bulletin also made it clear that certain types of information within this broad category 
did not have to be peer reviewed, including: 

• information distributed within the government or to contractors or 
grantees; 

• responses to requests under the Freedom of Information Act (FOIA) , 
the Privacy Act, or FACA;"^^ 

• correspondence with individuals; 

• press releases; 

• and distributions made to peer reviewers under the bulletin, provided 
they contain a disclaimer.^° 

The revised bulletin also explicitly exempted from its requirements information 
that (1) was related to national security, foreign affairs, or international trade or 
treaties where compliance with the bulletin “would interfere with the need for secrecy 



The explanatory section on requirements in the revised document states that the bulletin 
“addresses peer review of scientific information disseminations that contain findings or 
conclusions that represent the official position of one or more Departments or agencies of 
the federal government.” However, the term “official position” is not mentioned in the 
bulletin itself. The revised bulletin does not include the term “regulatory information.” The 
original proposed bulletin defines “study” as “any research report, data, finding, or other 
analysis.” While this definition is somewhat broader than the meaning usually ascribed to 
the term by scientists, it is narrower than tbe definition of “scientific information” in tbe 
revised bulletin, which includes “any communication or representation of knowledge ... in 
any medium or form.” 

FOIA is described in General Management Laws: A Compendium, coordinated by Clinton 
T. Brass, pp. 26-29. 

Tbe disclaimer is required to say “This information is distributed solely for tbe purpose 
of pre -dissemination peer review under applicable information quality guidelines. It bas not 
been formally disseminated by [tbe agency] and should not be construed to represent any 
agency determination or policy.” 
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or promptness’V' (2) was produced by government-funded scientists that was not 
represented as the views of the agency; (3) was an agency regulatory impact analysis 
or regulatory flexibility analysis under Executive Order 12866;^^ or (4) was medical, 
health, or safety information that was time sensitive or based on a recent clinical trial 
that had already been adequately peer reviewed. The bulletin also allowed agency 
heads to waive or defer the bulletin requirements “where warranted by a compelling 
rationale. 

In structure, the revised peer review bulletin was similar to the proposed bulletin 
in that it still essentially required agencies to take three actions (to the extent 
permitted by law): 

• have a peer review conducted on all “influential scientific 
information” that the agency intends to disseminate, 

• have all “highly influential scientific assessments” peer reviewed 
according to more specific and demanding standards, and 

• indicate what “influential” and “highly influential” information the 
agency plans to peer review in the future. 

Each of these required actions is described more fully in the sections below. 

Influential Scientific Information 

The revised peer review bulletin defined the term “influential scientific 
information” as information the agency “reasonably can determine that the 
dissemination of which will have or does have a clear and substantial impact on 
important public policies or private sector decisions.” Therefore, in contrast to the 
proposed bulletin, the peer review requirements in the revised bulletin were not 
limited to regulatory policies. 

In addition to the limitations and exemptions discussed above, the revised 
bulletin said that agencies were also not required to conduct a peer review of 
influential information that had already had an “adequate” peer review. The bulletin 
gave agencies substantial discretion in determining whether a prior review was 
“adequate,” specifically stating that the earlier review “need not comply with all of 
the requirements of this bulletin.” It also said that the sufficiency of the prior review 
depended on the “novelty and complexity of the science to be reviewed and the 
benefit and cost implications.” Eanguage in the proposed bulletin about rebutting an 



The proposed bulletin said agencies “need not have peer review conducted on significant 
regulatory information that relates to national defense or foreign affairs.” 

As noted previously, 0MB has said that the data used to generate these analyses are 
covered by the peer review requirements. Also, the bulletin did not indicate whether 
regulatory flexibility analyses under the Regulatory Flexibility Act were exempt. 

The proposed bulletin allowed 0MB (not the agencies) to waive many of the bulletin’s 
requirements if an agency “makes a compelling case” on grounds of an emergency, 
imminent bealtb hazard, homeland security threat, or “some other compelling rationale.” 
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agency ’ s presumption regarding whether earlier peer reviews were sufficient was not 
included in the revised bulletin. 

The revised bulletin (like the proposed bulletin) also gave agencies discretion 
in determining the appropriate peer review mechanism for any information not 
previously reviewed, with the level of rigor required again dependent on the “novelty 
and complexity of the science to be reviewed and the benefit and cost implications.” 
The bulletin stated that “appropriate” reviews could range from examinations by 
other federal specialists to formal panels composed of independent experts. The 
preamble to the bulletin indicated that some peer reviews could be done by letter 
(e.g., when a draft document covers only one discipline), but said panels were 
preferable when time and resources permitted their use. Peer reviewers were to be 
selected based on their relevant expertise, and were not to have participated in 
developing the product being reviewed. The revised bulletin also contained 
requirements to handle participation by scientists with conflicts of interest,^"^ and said 
that the comments of peer reviewers or a summary of those comments should be 
made available to the public. 

When Peer Review Is Required. In response to concerns that the trigger 
for peer reviews was too vague, 0MB indicated in the revised bulletin that the 
reviews were needed whenever an agency “reasonably determines” that scientific 
information “will have or does have a clear and substantial impact on important 
public policies or private sector decisions.” 

This change did not appear to address one of the underlying concerns voiced in 
the comments on the proposed bulletin. The definition of “influential” information 
is essentially the same as the one used to define “significant” information in the 
proposed bulletin; the only difference is the broader applicability to non-regulatory 
information. Therefore, agencies still may find it difficult to determine whether a 
particular study will, at some point in the future, have a “clear and substantial impact 
on important public policies,” much less “important private sector decisions.” Two 
agencies in analogous situations may reach different conclusions regarding the need 
for peer review. As a result, the overall purpose of the guidelines — a more 
consistent application of peer review among federal agencies — may not be achieved. 

Highly Influential Scientific Assessments 

The revised bulletin contained a number of additional peer review requirements 
for “scientific assessments” that are “highly influential.” These requirements were 
much more specific than those placed on “influential” scientific assessments, differed 
in some respects from the requirements in the proposed bulletin, and appeared to give 
0MB and the agencies significant authority to determine when the requirements are 
applicable. 



Specifically, the revised bulletin says that the agency (or the entity selecting peer 
reviewers) must (1) ensure that reviewers who are federal employees comply with federal 
ethics requirements, (2) apply or adapt those requirements to reviewers who are not federal 
employees, and (3) consider the National Academy of Sciences’ conflict of interest policy. 
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When Enhanced Peer Reviews Are Required. The revised bulletin 
defined a “scientific assessment” as “an evaluation of a body of scientific or technical 
knowledge which typically synthesizes multiple factual inputs, data, models, 
assumptions and/or applies best professional judgement to bridge uncertainties in the 
available information.” The term “highly influential scientific assessment” was 
defined in the revised bulletin as “influential” scientific information (as defined in 
the previous section of this report) that either the disseminating agency or 0MB 
determined 

• met the above-mentioned definition of a “scientific assessment”; and 

• could have a “clear and substantial impact on important public 
policies ... or private sector decisions with a potential effect of more 
than $500 million in any year,” or involved “precedent setting, 
novel, and complex approaches, or significant interagency interest.” 

As discussed previously, in the proposed bulletin enhanced peer review 
requirements applied to “especially significant” influential information expected to 
be disseminated in support of a regulatory action that could have a $100 million 
impact in any year. Therefore, although the dollar threshold for when enhanced peer 
review is required was higher in the revised bulletin ($500 million instead of $100 
million), the peer review requirements were more broadly applicable in that they 
applied to more than just regulations. Also, the language regarding “precedent 
setting, novel, and complex approaches” replaced criteria in the proposed bulletin in 
which 0MB alone could decide that certain information required enhanced peer 
review because it was of “significant interagency interest” or “relevant to an 
Administration policy priority.” 

Selection of Reviewers. The revised bulletin described in some detail how 
peer reviewers of highly influential scientific assessments should be selected, but still 
gave agencies substantial discretion in making the final decision. For example, the 
bulletin said that reviewers must be selected primarily on the basis of “necessary” 
expertise, experience, and skills, and must be diverse enough to “fairly” represent 
different perspectives. The revised bulletin also said that agencies must consider 
requesting the public to nominate reviewers, and must generally “bar participation 
by scientists employed by the agency sponsoring the review.” (As noted previously, 
the proposed bulletin said agencies should “strive” to select peer reviewers who were 
independent of the agency.) In addition, as in the proposed bulletin, the revised 
bulletin said agencies should try to avoid repeated use of the same reviewer on 
multiple assignments. 

However, in a significant departure from the proposed bulletin, the revised 
bulletin also said that research grants awarded to a scientist through a competitive, 
peer-reviewed process did not disqualify that scientist from serving on a peer review 
panel. (The proposed bulletin indicated that peer reviewers who were “receiving or 
seeking substantial funding from the agency through a contract or research grant” 
should not be selected as peer reviewers.) Also, whereas the proposed bulletin 
advocated balancing panel members’ biases, the revised bulletin contained no 
language to that effect. Finally, the revised bulletin said that agencies could 
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commission entities independent of the agency to select peer reviewers and manage 
the process. 

Public Comments. The revised bulletin also gave agencies discretion in the 
area of public comments. As noted previously, the proposed peer review bulletin 
instructed agencies to permit interested agencies and persons to submit comments on 
the information being reviewed, and to provide those comments to the peer reviewers 
for their consideration. In contrast, the revised bulletin said that agencies could 
decide on their own whether to make the information at issue available for public 
comment, but said agencies should “consider” having a public comment period and 
public meetings before the reviewers “when there is sufficient public interest.” The 
revised bulletin did not indicate how agencies are to make this determination, or how 
much public interest is “sufficient” to trigger the need for public comments. The 
bulletin also said that if an agency decides to make a draft assessment available to the 
public, it must, “whenever practicable,” provide peer reviewers with a summary or 
compilation of the comments received. 

Other Requirements. The revised bulletin contained several other types of 
requirements that were also in the proposed bulletin. For example, the revised 
bulletin required agencies to provide reviewers with “sufficient information” to allow 
them to understand the data, assumptions, and analytic procedures used to support 
key findings or conclusions. Also, agencies were required to instruct peer reviewers 
to prepare a report summarizing their findings and conclusions. The report must 
contain the names, affiliations, and credentials of the reviewers. Agencies were, in 
turn, required to prepare a written response to the report, and both the report and the 
response must be posted on the agency’s website. 

Alternative Procedures. The revised peer review bulletin said that an 
agency can ( 1 ) rely on scientific information produced by NAS, (2) commission NAS 
to peer review the information, or (3) use an alternative procedure “specifically 
approved by the [OIRA] Administrator, in consultation with OSTP.” As noted 
previously, the bulletin permitted the agency head to waive or defer some or all of the 
peer review requirements for “influential scientific assessments” and “highly 
influential scientific assessments” “where warranted by a compelling rationale.”^^ 
If the requirements are waived, however, the bulletin said peer review should be 
conducted “as soon as practicable thereafter.” The proposed bulletin did not provide 
for alternative procedures. 

0MB Authority. As is apparent from the preceding discussion, the revised 
peer review bulletin (like the proposed bulletin) gave 0MB significant authority to 
decide when agencies should use the more rigorous form of peer review. 
Specifically, the revised bulletin indicated that these procedures should be used when 
0MB (or the agency) determines that a scientific assessment (1) “could have a clear 
and substantial impact on important public policies ... or private sector decisions with 
a potential effect of more than $500 million in any year,” or (2) involves “precedent 
setting, novel, and complex approaches, or significant interagency interest.” 



The proposed bulletin would have permitted 0MB (not the agency heads) to waive the 
peer review requirements. 




http://wikileaks.org/wiki/CRS-RL32680 



CRS-22 



Therefore, under this definition, 0MB could unilaterally determine that certain 
information that an agency intends to disseminate is sufficiently “novel” or 
“complex” to trigger these “highly influential” additional peer review requirements.^^ 

Also, whereas the revised bulletin defined “influential”scientific information as 
information that the agency can reasonable determine “will have or does have” a 
clear and substantial impact on important public policies or private sector decisions, 
the bulletin said that information should be considered “highly influential” if either 
the agency or 0MB determines it “could” have such an impact — conceptually, a 
lower threshold than for information that is only “influential.” 

0MB retained significant authority over the peer review process in other ways. 
For example, 0MB could determine that an agency’s alternative peer review 
procedures are sufficiently rigorous to replace the requirements in the revised 
bulletin. 

Upcoming Peer Reviews 

The proposed bulletin required agencies to notify 0MB at least once each year 
of any existing or upcoming studies that might trigger the peer review requirements 
within the next year, and of the agencies’ plans for conducting those peer reviews. 
In contrast, the revised bulletin required each covered agency to post an “agenda” on 
its website every six months delineating any information disseminations subject to 
peer review. The revised bulletin was also much more specific about these notices 
than the proposed bulletin, requiring each entry on the agenda to contain a detailed 
description of the “peer review plan,” including: 

• the title, subject and purpose of the plan, with an agency contact, 

• how the review will be conducted (panel or individual letters), 

• the anticipated number of reviewers, and 

• a “succinct description of the primary disciplines or types of 
expertise needed in the review.” 

Each peer review plan was also required to describe how reviewers will be selected, 
whether the public will be asked to nominate peer reviewers, whether there will be 
opportunities for the public to comment on the information being reviewed, and 
whether the agency will provide reviewers with copies of any public comments 
before beginning their work. The revised bulletin required agencies to establish a 
mechanism to allow the public to comment on the adequacy of the peer review plans, 
and must consider those comments. 



In the proposed bulletin, 0MB had similar authority. The agency could be required to 
conduct a formal, independent, external peer review if 0MB determined that the information 
was of “significant interagency interest or [was] relevant to an Administration policy 
priority.” 




http://wikileaks.org/wiki/CRS-RL32680 



CRS-23 



Comments on the Revised Bulletin 

OMB said that 57 individuals and organizations commented on the April 2004 
revised peer review bulletin.^^ About a dozen of the comments essentially asked 
OMB to allow more time for public comments and interagency review. Most of the 
substantive comments provided were supportive of at least some of the changes that 
OMB made to the bulletin. However, some commenters believed the changes had 
weakened the bulletin to such an extent that they withdrew their initial support, while 
others believed the changes had not gone far enough. Some of the individuals’ and 
organizations’ comments are summarized below. 

National Academy of Sciences 

The presidents of the National Academy of Sciences and its two affiliated 
institutions, the National Academy of Engineering and the Institute of Medicine, 
released a statement on April 15, 2004 that generally praised the revision, arguing 
that it would “improve the quality of the government’s scientific assessments and ... 
decision-making,” and “better accommodate the diverse circumstances of ... federal 
agencies. The statement suggested, but did not state, that further improvements 
were nevertheless desirable. 

OMB Watch 

In May 2004, OMB Watch filed a comment on the revised peer review bulletin, 
stating that although OMB had made “significant revisions” to the bulletin, it 
nevertheless had “fail[ed] to correct some of the most fundamental complaints. “ 
Specifically, OMB Watch said that: 

• OMB lacked the authority to issue the guidelines, given (1) the lack 
of specific authority to do so in the statutes and orders cited and (2) 
that Congress had not enacted legislation designed to require peer 
review. 

• OMB had not established that there was a government- wide peer 
review “problem” in need of a government-wide solution. 



For OMB’s response to the comments provided on the revised peer review bulletin, see 
[http://www.whitehouse.gOv/omb/inforeg/peer2004/peer_response.pdf] . For a link to each 
of the comments, see [http://www.whitehouse.gov/omb/inforeg/peer2004/list_peer2004. 
html]. 

National Academy of Sciences, “Academies’ Presidents Comment on OMB Peer Review 
Guidelines,” Press Release, Apr. 15, 2004, at [http://www4.nationalacademies.org/news.nsf/ 
isbn/s04 1 52004?OpenDocument] . 

“...the extended comment period will permit further refinements in the proposed 
guidance...” (ibid.). 

“ For a copy of OMB Watch’ s comments, see [http://www.ombwatch.org/info/dataquality/ 
CommentsRevisedPR-OMBW.pdf] . 




http://wikileaks.org/wiki/CRS-RL32680 



CRS-24 



• 0MB retained too much control over federal peer review practices 
(e.g., in determining when certain information requires a more 
rigorous peer review and in approving alternative peer review plans), 
and recommends that some of those responsibilities be given to the 
National Academy of Sciences. 

• the “independence” standard in the bulletin would diminish the 
ability of government scientists to participate as peer reviewers, 
while not affecting scientists from industry. 

• the revised bulletin would likely significantly delay the regulatory 
process at agencies by adding “several new layers of bureaucracy” 

(e.g., public comments on peer reviews and peer review plans). 

In its conclusion, 0MB Watch recommended that 0MB withdraw the bulletin and 
instead convene an interagency committee to investigate peer review techniques and 
flaws, or make changes in the bulletin to address their concerns. 

Chamber of Commerce 

The U.S. Chamber of Commerce submitted comments on the revised bulletin 
in May 2004, stating that the revised bulletin has “fundamental problems that must 
be resolved,” and that it could not support the issuance of a final bulletin until those 
problems are addressed.'’^ The Chamber said it had “deep concerns” about (among 
other things): 

• the blanket exemption given to NAS peer reviews; 

• the “excessive” discretion given to agencies in the implementation 
of the bulletin; 

• the absence of “provisions allowing affected parties to contest any 
agency determination of applicability, peer review type, panel 
selection, charge, or other peer review program element as it applies 
to a specific case;” and 

• insufficient limitations on the use of proprietary information and 
models. 

In summary, the Chamber said that the changes 0MB made in issuing the revised 
bulletin “are so severe and debilitating as to eliminate the public benefit of having a 
common, government-wide minimum standard for peer review.” 



For a copy of the Chamber’s comments, see 

[http://www.uschamber.com/NR/rdonlyres/eznlbdbxz7wylilw7now26n2cqigrnx57hqpav 
qkxf7xgvfy2foiy5w5bd37suvonnhln7jv7myxop/COMMENTSProposedRevisedBulletino 
nPeerReviewandInfo.pdf] . 
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Members of Congress 

In May 2004, twelve Members of Congress provided 0MB with comments on 
the revised bulletin, stating that the revision did not address previously expressed 
concerns that the proposal was “unjustified, overly broad, burdensome, and did not 
appropriately guard against appointment of reviewers with conflicts of interest,” or 
that it provided 0MB with “excessive authority over the production and 
dissemination of government information.®^ The Members said that the need for the 
bulletin remained unjustified, that the exemptions in the revised bulletin resulted in 
an unbalanced approach, and that it created “considerable new burdens on agencies.” 
They also said that “safeguards to ensure the integrity of the process for selection of 
reviewers have not been included in the Bulletin.” In particular, they said there was 
“no requirement that agencies disclose publicly whether any panel members with a 
conflict were selected.” In conclusion, the Members characterized the bulletin as “a 
wolf in sheep’s clothing,” and said it was “designed to prevent the dissemination of 
government information, stifle public debate, and delay legitimate government 
regulatory action.” They also said that the agencies, not 0MB, are “the repository 
of scientific and technical expertise,” and said they should be given “full authority 
to govern their own peer review procedures.” 

AAAS 

AAAS filed a comment with 0MB in June 2004, saying that the “revised 
version is much improved” but raising several issues.®^ For example, AAAS: 

• questioned why 0MB said scientists seeking an exemption to the 
peer review requirements for information produced by government- 
funded scientists were only “advised” to include a disclaimer rather 
than required to do so; 

• suggested that agencies be required to make public the criteria for 
determining when outside entities would be commissioned to select 
peer reviewers or manage the peer review process, and how these 
entities would be selected and overseen by the agencies; and 

• endorsed the idea of an interagency working group on peer review, 
and recommended that the group be required to report to the public 
annually on its deliberations and findings, and to conduct studies and 
hold meetings regarding the impact of the guidelines on science and 
the quality of the information produced. 

The editor of AAAS’s flagship publication. Science, praised 0MB for its efforts to 
address the concerns of the scientific community, saying that most of the concerns 



For a copy of these Members’ comments, see [http://www.whitehouse.gov/omb/inforeg/ 
peer2004/25.pdf]. 

®^ For a copy of tbe AAAS comments, see [http://www.aaas.org/spp/cstc/docs/5-28-04OMB 
peerltr.pdf]. 
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regarding the proposed bulletin were answered in the revision, and that 0MB 
“deserves credit for being a good listener.”®'^ 

Other Scientific Societies 

A number of other scientific societies also commented on the revised peer 
review bulletin. For example, an official in the Association of American Medical 
Colleges was quoted in April 2004 as saying that 0MB had been “very responsive,” 
and said the revised document “is much, much more flexible.”^^ In July 2004 (after 
the comment period for the revised bulletin closed), a consortium of 13 scientific 
societies, mostly in the life sciences, released a general statement on the role of 
scientific peer review in policy development.^^ The statement cited peer reviews as 
“critically important tools for policy makers” and laid out a list of eight 
“considerations” the societies believe will help ensure that agency peer reviews are 
well designed and performed. While several of the points were also made by other 
organizations, some provided a somewhat different emphasis. For example, the 
societies: 

• stressed that for scientific peer review to be effective, it “should be 
insulated from politics as much as possible.” 

• said peer review is not a way to get the “right” answer for policy; in 
many instances, even the best data are subject to alternative 
interpretations, and there is no “best” answer. 

• said that, while avoiding and managing conflicts of interests is 
important, most scientific peer review rests on the presumed 
integrity of the reviewers, stemming from professional standards of 
conduct to which scientists subscribe. 

• said it is critical to acknowledge the cultural differences among 
scientists, policymakers, and the public in developing and 
implementing an appropriate model for peer review. For example, 
science is inherently uncertain, and scientists must acknowledge that 
in their work, including peer review. Managers, in contrast, often 
are not in a position to embrace that uncertainty but must make 
decisions in the face of it. 



Donald Kennedy, “Praise, for a Change,” Science, vol. 304 (May 21, 2004), p. 1077. 

“White House Softens Disputed Peer-Review Plan,” Science, vol. 304 (Apr. 23, 2004), 
p. 496. 

American College of Preventive Medicine et al., “Scientific Peer Review in Policy 
Making,” Position Statement, July 15, 2004, [http://www.aibs.org/position-statements/ 
0407 1 5_scientific_peer_.html] . 
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Final Peer Review Bulletin 

As noted previously, on December 15, 2004, 0MB released its final bulletin on 
peer review. 0MB said that it had made “minor revisions” to the April version of the 
bulletin that were “responsive to the public’s comments.”^^ Perhaps most notably, 
0MB said that the final bulletin: 

• requires that agencies disclose to the public the names of peer 
reviewers for both “influential scientific information” and “highly 
influential scientific assessments.” However, the bulletin does not 
require agencies to disclose which reviewers made which comments. 

• adds an annual reporting requirement to address concerns about the 
lack of an enforcement mechanism, thereby allowing 0MB to “track 
how agencies are using the Bulletin, including provisions for 
waivers and exceptions.” 0MB also said it expects the public to 
monitor agencies’ actions under the bulletin. 

• requires agencies to designate at the time they plan a peer review 
whether it should be considered “influential scientific information” 
or a “highly influential scientific assessment.” 

• provides additional criteria for what constitutes a “highly influential 
scientific assessment.” Specifically, the term now includes 
influential assessments that are “novel, controversial, or precedent 
setting or have significant interagency interest.” 0MB also clarified 
that the $500 million impact test covers all impacts (public or 
private sector) that occur in any year. 

• broadened the applicability of the exemption for time-sensitive 
health or safety information, noting that it is not restricted to medical 
data from clinical trials that were subject to adequate peer review 
before the start of the trial. 

• requires agencies to adopt or adapt the National Academy of 
Sciences policy for committee selection with respect to evaluating 
conflicts of interests of peer reviewers who are not federal 
employees. 

0MB also reportedly made a number of what appear to be less significant changes 
to the final bulletin. For example, 0MB said that the final version of the bulletin: 

• retains a substantial degree of discretion for the agencies, but also 
clarifies the intent of its language concerning the “adequacy of prior 



The characterizations in this section of 0MB ’s changes in response to public comments 
are from 0MB ’s response to the comments provided on the revised peer review bulletin, 
available at [http://www.whitehouse.gov/omb/inforeg/peer2004/peer_response.pdf] . 
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peer review” and the use of deferrals and waivers to address 
concerns that the revised draft was too discretionary. 

• encourages agencies to provide public participation opportunities for 
reviews of “highly influential assessments,” but avoids a strict 
mandate for public participation and adds language “to stress that 
agencies should avoid open-ended comment periods.” 

• “reinstate[s]” instructions that peer reviewers are to comment on 
scientific and technical questions, and not to provide opinions on the 
policy implications. 

• provides a “rare exception” to allow a premier government scientist 
from the department or agency that prepared an assessment to 
participate in a peer review (under certain circumstances). 



Concluding Observations 

OMB’s peer review bulletin is likely to have a significant effect on federal 
rulemaking and other forms of information dissemination and public policy. That 
effect is likely to be both direct (through agencies’ and OMB’s enforcement of the 
bulletin’s requirements) and indirect (through references to the bulletin by others). 
For example. Section 402 of the Specialty Crops Competitiveness Act of 2004 (P.L. 
108-465, signed by the President on December 21, 2004) indicated that a required 
peer review of the procedures and standards governing the consideration of certain 
import and export requests “shall be consistent with the guidance by the Office of 
Management and Budget pertaining to peer review and information quality.” 

Some of the initial issues and concerns raised by commenters on the proposed 
peer review bulletin were clarified or otherwise addressed in the revised and final 
versions of the bulletin. Perhaps most notably, the bulletin now makes it clear that 
scientists are not prohibited from serving as peer reviewers if they receive research 
grants from the agency based on investigator-initiated, competitive, peer reviewed 
proposals . The bulletin also provides exemptions from the peer review requirements 
for time-sensitive medical, public health, and safety information, and other 
compelling circumstances. However, a number of other issues remain unclear, 
including (1) the amount of discretion that agencies will actually have in carrying out 
the bulletin’s requirements, (2) the degree to which the bulletin will accomplish the 
stated goal of making federal peer review practices more consistent, and (3) the 
effects of the bulletin on agencies and the federal rulemaking process. 

Agency Discretion 

The final peer review bulletin appears to give federal agencies substantial 
discretion in determining whether peer review is required for specific products and, 
if so, what type of peer review mechanism is appropriate and who should serve as 
peer reviewers. For example, the bulletin says agencies need not have peer review 
conducted on influential scientific information that had already been subject to 
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“adequate” peer review. To determine whether a prior review was “adequate,” 
agencies are directed to consider (among other things) whether the science is “novel” 
or “complex,” and whether it is “important” to decision making. (Notably, though, 
the final bulletin no longer indicates that an “adequate” peer review “need not 
comply with” all of the requirements of the bulletin.) Also, the final bulletin says 
that information should be considered “influential scientific information” if the 
agency “reasonably can determine” that it will have or does have a “clear and 
substantial” impact on “important” public policies or private sector decisions. The 
bulletin says that peer reviewers should be selected to provide the necessary 
expertise, experience, and skills, and the group of reviewers should be “sufficiently” 
broad and diverse to “fairly” represent the “relevant” scientific perspectives and 
knowledge. 

On the other hand, the final bulletin also gives 0MB substantial discretion in 
certain areas. For example, the bulletin indicates that 0MB can require agencies to 
use the more exacting procedures for “highly influential scientific assessments” if 
0MB determines the information is a “scientific assessment” that “could” have a 
substantial impact on public policies or private sector decisions with a “potential 
impact” of more than $500 million in any year, or is “novel, controversial, or 
precedent setting,” or has “significant” interagency interest. Also, 0MB can 
unilaterally approve agencies’ use of alternative peer review procedures. 

The amount of discretion that agencies will actually have in carrying out their 
peer review programs (or, conversely, the amount of control that 0MB will retain) 
will be apparent only through the bulletin’s implementation. The amount of agency 
discretion (or 0MB control) could vary substantially from one administration to 
another. 

Consistency 

OMB indicated that stronger peer review policies were needed because of the 
importance of the issue and, citing a 1999 GAO report, because of the “variability in 
both the definition and implementation of peer review across agencies.” OMB went 
on to say that, prior to the development of the bulletin, “there were no government- 
wide standards concerning when peer review is required and, if required, what type 
of peer review processes are appropriate.” Therefore, OMB said that the bulletin 
“establishes minimum standards for when peer review is required for scientific 
information, and the types of peer review that should be considered by agencies in 
different circumstances.” 

However, the final bulletin may not provide the desired consistency in peer 
review definition or implementation. As indicated above, the bulletin leaves many 
key terms undefined or subject to interpretation, and gives the agencies substantial 
discretion regarding when certain actions should be taken (e.g., when previous peer 
reviews are “adequate”) and which reviewers should be selected (i.e., those with the 
“necessary” expertise, experience, and skills). To the extent that agencies are 
allowed to exercise discretion in these areas, consistency may be forfeited. On the 
other hand, strict enforcement of uniform procedures established by OMB carries 
with it a different set of concerns about aggregation of power within the Executive 
Office of the President, and may be resisted by federal agencies. Also, as noted 
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previously, although GAO reported that agencies’ peer review practices were 
inconsistent, it did not recommend greater consistency, and some view variation in 
those practices as appropriate and desirable. 

Effects on Agencies and Rulemaking 

A number of commenters expressed concerns regarding the effect that adding 
new peer review requirements would have on what is already viewed by some 
observers as a lengthy, “ossified” federal rulemaking process. Somewhat related 
concerns have been voiced regarding the cost of the requirements to federal agencies, 
with Senators Lieberman and Durbin suggesting that the requirements “do not pass 
0MB ’s own [cost-benefit] test of good rulemaking.” In response, 0MB provided 
estimates in the preamble to the revised bulletin indicating that it did not believe the 
costs would be prohibitive to the agencies. As noted previously, though, in 1997, 
0MB indicated that peer review costs could be significant in terms of both time and 
agency resources.^* Little empirical data are currently available regarding the cost of 
peer reviews, how they affect the federal rulemaking process, or their effect on the 
quality of the information being reviewed. It is even less clear how the peer review 
requirements suggested by 0MB will affect those factors, and there appears to be no 
mechanism in place for collecting such data. There are some indications, however, 
that the requirements could delay regulatory action.®^ 

The AEI-Brookings Joint Center for Regulatory Studies recommended that 
0MB or some other entity build into the peer review program an evaluation to 
determine its effect on the quality of regulatory analyses. Any such evaluation could, 
at least conceptually, include an examination of the cost of the new peer review 
requirements to federal agencies and its effect on the pace of rulemaking. To 
determine the effect of the peer review bulletin on these or other factors, baseline 
information regarding the current state of the art would need to be gathered before 
the bulletin’s implementation. 

Even if the data indicate that peer review adds time to the early stages of the 
rulemaking process, that time may be worth the investment if doing so reduces the 
likelihood of subsequent judicial challenges to the rules. Peer review may also 
provide agencies with a preview of likely objections to a rule during the notice and 
comment phase, thereby allowing them to minimize any weaknesses and respond 
quickly to adverse comments. All of these factors would have to be considered in 
any evaluation of the effect of peer review on the regulatory process. 



Statement of Sally Katzen, OIRA Administrator, before the Senate Committee on 
Governmental Affairs, Sept. 12, 1997. 

Ben Geman, “White House Peer Review Requirements Could Delay Standards,” 
Greenwire, volume 10, number 9 (Dec. 17, 2004). The article quoted a program manager 
in the Department of Energy as saying that efficiency standards for residential furnaces and 
boilers, commercial air conditioners and other equipment were being delayed for two years 
because of 0MB ’s peer review bulletin. 
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Table 1. Comparison of Proposed, Revised, and Final Peer Review Bulletins 

Issue Proposed Bulletin Revised Bulletin Final Bulletin 

Target of basic level of peer “Significant regulatory information,” “Influential scientific information,” defined Same as revised bulletin, 

review defined as scientific or technical as scientific information (including but not 

regulatory information that the agency limited to regulatory information) that the 
can “reasonably determine” will have, agency “reasonably can determine”will 
when disseminated, a “clear and have, when disseminated, “a clear and 

substantial impact on important public substantial impact on important public 
ipolicies or important private sector policies or private sector decisions.” 

decisions.” 

^ 

Target of more rigorous peer ^Especially significant regulatory “Highly influential scientific assessments,” Essentially the same as revised 

review information,” defined as significant defined as influential scientific information bulletin. “Highly influential 

Regulatory information (1) disseminated that either the agency or 0MB determines scientific assessments” defined as 

jin support of an economically to be “scientific assessments” (e.g., state- influential scientific information 

^significant regulation (e.g., $100 million of-science reports, meta-analyses, or risk that the agency or 0MB determine 

lannual impact on the economy); (2) that assessments) that (1) could have a “clear (1) could have a “potential impact” 

bcould, when disseminated, have a “clear and substantial impact on important public (changed from “clear and substantial 

~and substantial impact on important policies or private sector decisions with a impact”) of more than $500 mi llion 

public policies or important private potential effect of more an $500 million in in any year or (2) are “novel, 

sector decisions” (those with $100 any year; or (2) involve “precedent setting, controversial [changed from 

million impact in any year); or (3) 0MB novel, and complex approaches, or “complex”]or precedent-setting” 

determines is of “significant interagency significant interagency interest.” scientific assessments (changed 

interest” or “is relevant to an from “approaches”). 

Administration policy priority.” 
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Issue 


Proposed Bulletin 


Revised Bulletin 


Final Bulletin 


Selection of peer reviewers 
(independence and balance) 


Peer reviewers “should” not have 
participated in the development of the 
work product. 


Same as proposed bulletin. 


Reviewers “shall” not have 
participated in development of work 
product. 




Agency “shall strive to appoint experts 
who ... are independent of the agency, 
[and] do not possess real or perceived 
conflicts of interest.” One of the factors 
|to consider in determining independence 
^s whether the individual “is currently 
Receiving or seeking substantial funding 
afrom the agency through a contract or 
Sresearch grant.” 

'bC 

q 

:/i 

a 

O 


Agency (or entity selecting the reviewers) 
must “bar participation by scientists 
employed by the agency sponsoring the 
review unless the reviewer’ s service as a 
peer reviewer defines the government 
employment (i.e., special government 
employees).” However, “[rjesearch grants 
that were awarded to scientists based on 
investigator-initiated, competitive, peer- 
reviewed proposals generally do not raise 
issues as to independence or conflicts.” 


Essentially the same as revised, but 
adds exception for the “rare case” 
where the agency determines that a 
premier government scientist (1) is 
employed in a different agency of a 
Cabinet-level department, (2) is not 
in a position of management or 
policy responsibility, and (3) 
possesses “essential expertise that 
cannot be obtained elsewhere.” 




|Also consider whether the individual has 
previously “advocated a position on the 
specific matter at issue.” 


No similar provision. 


No similar provision. 




Also consider whether the individual has 
conducted multiple peer reviews for the 
same agency in recent years. 


Agencies “should avoid repeated use of the 
same reviewer on multiple assignments 
unless his or her participation is essential.” 


Agencies “shall” avoid repeated use 
of the same reviewer unless 
participation is essential “and cannot 
be obtained elsewhere.” 




If obtaining the appropriate expertise 
requires selecting a reviewer with an 
apparent bias, agencies should ensure 
that another reviewer with a contrary 
bias is selected. 


No similar provision. 


No similar provision. 
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Issue 


Proposed Bulletin 


Revised Bulletin 


Final Bulletin 




Agencies should avoid repeated use of 
the same reviewer on multiple 
assignments. 






Selection of peer review 
mechanism (significant or 
influential information) 


Agencies should select an appropriate 
peer review mechanism (e.g., review 
within agency or formal review by 
independent body) based on the novelty 
=and complexity of the science to be 
Reviewed, the benefit and cost 
implications, and any controversy 
^regarding the science. 


Same as proposed bulletin. 


Essentially the same criteria, but 
also (1) the importance of the 
information to decisionmakers, (2) 
the extent of prior peer review, and 
(3) the factors regarding 
transparency (see below). 


Public comments 


s 

^Agencies are required to “provide an 
Opportunity for other interested agencies 
|and persons to submit comments,” and 
imust “ensure that such comments are 
^provided to the peer reviewers with 
Sample time for consideration before the 
peer reviewers conclude their review 
and prepare their report.” 


Agencies are allowed (but not required) to 
obtain comments on draft scientific 
assessments. If they do so, they must 
(“whenever practical”) provide a summary 
or compilation of those comments that 
address “significant scientific and technical 
issues.” When there is “sufficient public 
interest,” agencies must “consider” 
establishing a public comment period on 
the peer reviewers’ draft report and 
sponsoring a public meeting. 


“Whenever feasible and 
appropriate,” agencies are required 
to make the draft scientific 
assessment available to the public 
for comment when submitted for 
peer review and to sponsor a public 
meeting. If they do so, agencies 
must (“whenever practical”) provide 
peer reviewers with access to the 
public comments. 


Alternatives to the bulletin’s 
procedures 


No comparable language 


Agencies may (1) rely on scientific 
information produced by NAS, (2) 
commission NAS to peer review the 
information, or (3) use an another 


Same as revised bulletin. 



procedure specifically approved by 0MB. 
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Issue 



Transparency requirements 



Upcoming peer reviews 



Proposed Bulletin 


Revised Bulletin 


Final Bulletin 


The agency shall direct peer reviewers 
(“individually or often as a group”) to 
issue a final report “detailing the nature 
of their review and their findings and 
conclusions.” The report should also 
disclose “the names, organizational 
affiliations, and qualifications of all peer 
reviewers, as well as any current or 
Iprevious involvement hy a peer reviewer 
^ith the agency or issue under peer 
Review consideration.” The report 
o“should he included in the 
^administrative record for any related 
Rulemakings.” 

q 

a 

O 


A detailed summary or copy of the 
reviewers’ comments, as a group or 
individually, shall he made available to the 
public.” Also, the report shall be part of 
the administrative record for related agency 
actions “where appropriate.” 


The agency or entity managing the 
peer review shall instruct reviewers 
to prepare a report describing “the 
nature of their review and their 
findings and conclusions.” The 
report must (1) either include a 
verbatim copy of each reviewer’ s 
comments (with or without 
attribution) or represent the views of 
the group as a whole, and (2) 
disclose the names of the reviewers 
and their organizational affiliations. 
The report must be disseminated on 
the agency’s website, discussed in 
any related rulemaking, and 
included in the administrative 
record. 


a 

^Agencies shall provide 0MB at least 
once each year (1) a description of any 
existing or upcoming studies that might 
constitute or support significant 
regulatory information that the agency 
intends to distribute within the next 
year, and (2) the agency’s plan for 
conducting a peer review of such 
studies. 


Agencies shall post on their websites, 
updated at least every six months, an 
agenda designating all planned and ongoing 
information disseminations subject to peer 
review. For each entry, agencies must 
provide detailed information in a “peer 
review plan” (e.g., the subject and purpose 
of the review, how the review will be 
conducted, the number of reviewers, how 
they will be selected, whether the public 
will be allowed to comment, and whether 
reviewers will be provided with those 
comments). Agencies are also required to 
allow the public to comment on the plan, 
and to consider those comments. 


Same as revised, but agencies also 
required to provide a link from the 
agenda to each document made 
public pursuant to the bulletin. 

Also, agencies’ peer review plans 
should indicate whether the 
dissemination will be “influential” 
or “highly influential,” the timing of 
the review, and whether alternative 
procedures will be employed. 
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Issue 


Proposed Bulletin 


Revised Bulletin 


Final Bulletin 


Exclusions 


Peer reviews are not required for: 


Peer review is not required for: 


Same as revised bulletin, except: 




— studies that have already had an 


— information produced hy government- 


— time sensitive “medical” 




“adequate” peer review (with 


funded scientists if not represented as 


dissemination not excluded 




review hy a scientific journal 


the views of the agency (with 


(only “health or safety”); 




considered adequate, hut 


disclaimer); 


— underlying data and analytical 




rehuttahle); 


— information related to national 


models used in regulatory 




— regulatory information related to 


security, foreign affairs, or 


impact analyses or flexibility 




national defense or foreign affairs; 


negotiations involving international 


analyses are not excluded; and 




1 


trade or treaties; 


— an exclusion added for 




2 — information disseminated in the 


— information disseminated in the 


analyses of routine statistical 




^ course of an adjudication or permit 

6 application. 

'b£ 

q 

c/2 

o 


course of an adjudication or permit 
application, unless the agency 
determines it is novel or precedent 
setting; 

— medical, health, or safety 
dissemination that is time sensitive or 
based on clinical trial data that was 
previously peer reviewed; 

— regulatory impact analyses or 
flexibility analyses subject to 
interagency review under E012866; 
or 

— routine statistical information. 


information. 


Waivers and deferrals 


OMB may waive the peer review 
requirements if agency makes a 
“compelling case” that waiver is needed 
because of: 

— an emergency, 

— an imminent health hazard, 

— a homeland security threat, 

— “some other compelling rationale” 


The agency head may waive or defer some 
or all of the peer review requirements for 
“influential” and “highly influential” 
information/assessments “where warranted 
by a compelling rationale.” If deferred, the 
review should be conducted “as soon as 
practicable thereafter.” 


Same as revised bulletin. 
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Issue 


Proposed Bulletin 


Revised Bulletin 


Final Bulletin 


Annual reports 


No provision. 


No provision. 


By December 15, agencies required 
to provide 0MB with a report 
containing (1) the number of peer 
reviews conducted during the fiscal 
year; (2) the number of times 
alternative procedures were 
invoked; (3) the number of times 
waivers or deferrals were invoked; 




X 

O 




(4) any decisions to use exceptions 




Cvl 




in reviewer appointments; (5) the 




2 




number of panels conducted in 




e: 

o 




public and allowed public 








comments; (6) the number of public 




'b£ 




comments provided on review 




q 

c/2 




plans; and (7) the number of 




o 




reviewers recommended by 








professional societies. 



& 













